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Abstract 
It is well established that financial aid, in the form of grants, increases the probability of 
enrollment in postsecondary education. A slate of studies in recent years has extended this 
research to examine whether grant aid also has an impact on persistence and degree attainment. 
This paper presents a systematic review and meta-analysis of the causal evidence of the effect of 
grant aid on postsecondary persistence and degree attainment. A meta-analysis of 42 studies 
yielding 73 effect sizes estimates that grant aid increases the probability of student persistence 
and degree completion between two and three percentage points, and estimates that an additional 
$1,000 of grant aid improves year-to-year persistence by 1.2 percentage points. Suggestions for 


future research and implications for policy are discussed. 
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The Effects of Grant Aid on Student Persistence and Degree Attainment: 
A Systematic Review and Meta-Analysis of the Causal Evidence 

The advent of widely accessible financial aid programs to support postsecondary study 
has been a major factor in the democratization of higher education in the United States. Nearly 
half (7.8 million) of the 16 million returning World War II veterans took advantage of GI Bill 
funds to pursue an education or training program by 1956 (U.S. Department of Veterans Affairs, 
n.d.). With the authorization of the Higher Education Act (HEA) of 1965, financial aid cemented 
its role as a federal policy tool to be wielded for the creation of a skilled labor force and a 
democratic citizenry. In the intervening years, individual states and institutions have adopted 
both need-based and merit-based grant aid programs to supplement federal affordability efforts. 
Where a college or university education had once been accessible only to the affluent, the wide 
availability of grants put postsecondary education within reach for middle- and working-class 
families. 

As the number of Americans pursuing undergraduate education has grown, from 9.5 
million in 1976 to 19 million in 2015 (NCES, 2016), receipt of financial aid to support college 
financing has increasingly become the norm. In academic year 2014-2015, 86 and 79 percent of 
first-time, full-time degree/certificate-seeking undergraduate students at 4-year and 2-year 
degree-granting postsecondary institutions, respectively, were awarded financial aid (McFarland 
et al., 2017). 

Much of this substantial investment in subsidizing postsecondary education is in the form 
of grants. Today, over $125 billion annually flow to postsecondary students in the form of grant 
aid, with more than a third of that sum coming from the federal government (College Board, 


2016). Need-based grants may be justified exclusively on economic equity grounds (see Baum 
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(2007) for an application of Rawlsian justice to financial aid). However, there is also substantial 
empirical evidence augmenting the equity case. Numerous studies have identified a positive 
relationship between both need-based and merit-based grant aid and student outcomes including 
college enrollment, academic performance, persistence, and degree attainment (e.g., Angrist, 
Oreopoulos, & Williams, 2014; Deming & Dynarski, 2009). A subset of those studies have 
employed experimental and quasi-experimental causal estimation methods which provide the 
most accurate estimates available, and these studies have confirmed the important role financial 
aid plays in the postsecondary access and success of undergraduate students. 

Decisions regarding the prioritization of scarce dollars for student financial support 
should be based on a body of high-quality empirical work. Several older attempts to review the 
literature on the impacts of financial aid exist (Jensen, 1983; Leslie & Brinkman, 1988; St. John, 
1991), but, in addition to being outdated, these studies incorrectly considered observational data 
and analyses as evidence of causal impacts. The application of causal quantitative analytic 
methods in the field of educational research has improved tremendously over the last few 
decades, and a modern review of the literature of the efficacy of grant aid would preference the 
inclusion of well-identified causal studies over observational studies prone to omitted variables 
bias. 

One recent review of the literature did account for the differences in observational, quasi- 
experimental, and experimental analyses (Deming & Dynarski, 2009). After reviewing the causal 
literature, Deming and Dynarski (2009) found a reduction in college costs increased 
student enrollment, although the estimates varied substantially across studies. They concluded 
reducing costs by $1,000 increases the likelihood of enrollment by about four percentage points. 


Until very recently, the majority of causal research on the efficacy of financial aid has focused 
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on the effects of grant aid on college enrollment, hence their literature review necessarily 
concentrates on enrollment outcomes. 

While measuring the effect of subsidies on college enrollment is crucial, the educational 
goal of individuals and society is often degree attainment. It is possible that grant aid induces 
students to enroll in higher education but does not help them persist or graduate. Descriptively, 
college enrollment has grown steadily from the mid-1990s to present, but degree completion has 
lagged behind (Lumina Foundation, 2017). While there is evidence of the economic returns to 
even just a year of college (Kane & Rouse, 1999), there is a substantial earnings premium to 
completing a bachelor’s degree (College Board, 2016). 

Fortunately, recent years have produced a growing body of research on the causal effects 
of grant aid on persistence and degree attainment, albeit with mixed results. While many studies 
found positive effects of financial aid on persistence and degree attainment (e.g., Angrist, Autor, 
Hudson, & Palais, 2014; Bettinger, 2015; Scrivener et al., 2015), there are other studies that 
found null or even negative results (e.g., Clotfelter, Hemelt, & Ladd, 2016; Cohodes & 
Goodman, 2014; Partridge, 2013). While the body of literature examining the causal effects of 
grant aid continues to expand, there is neither a systematic review of that literature nor a meta- 
analysis that can assess the overall impact of aid on student persistence and degree completion 
across these varied studies. Our systematic review and meta-analysis attempt to fill that gap in 
the literature. 

Moreover, the results of this meta-analysis are of interest for policymakers and 
researchers. Increasing college affordability remains a top policy priority at the state and federal 
levels, particularly as public concerns about the affordability of college and the consequences of 


student debt continue to grow. Early indicators of the potential for financial aid to influence 
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persistence and student success have contributed to the rapid proliferation and growth of 
financial aid programs. A number of states, including New York, Oregon, and Tennessee, are 
undertaking considerable state-level grant aid programs to ensure that subsets of the 
undergraduate population can attend college free of tuition and fees. A systematic review and 
estimation of the causal effect of grant aid on student persistence and degree attainment presents 
the opportunity to reflect on the likely outcomes of such programs, and our analysis also sheds 
light on how much aid is necessary to achieve an effect on persistence and degree attainment. 
Whether grant aid plays a role in persistence also has implications institutional practice. 
Research on grant aid’s impact on persistence outcomes informs whether “front loading,” a 
process in which substantial grant aid is provided in the first year of college but withdrawn in 
subsequent years, is an effective method of allocating limited grant dollars (Avery & Hoxby, 
2004; Hossler, Ziskin, Gross, Kim, & Cekic, 2009). 
To provide policymakers and practitioners with insight into these important issues, our 
study answers three research questions: 
di What is the causal effect of grant aid on student persistence and degree attainment 
across extant studies that provide causal estimates? 
2 Do the causal effects of grant aid on student persistence and degree attainment 
vary across study and program characteristics? 
3 What is the estimated effect of an additional $1,000 of grant aid on persistence 
and degree attainment? 
Our study contributes to the literature by providing the first systematic review of the 
causal estimates of grant aid on student persistence and degree attainment. Our meta- 


analysis effect estimates distill the calculated effects from forty-two U.S. studies and five 
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international studies and assess how those effect estimates vary by design of the grant aid 
program and quality of the research method. Our analysis enables us to provide an effect of 
$1,000 of grant aid on persistence similar to the effects widely cited on enrollment (Deming 

& Dynarski, 2009). Overall we find that grant aid programs increase the probability of persisting 
and degree completion from two to three percentage points, and, assuming a linear relationship 
of aid amount and impact, we estimate that an additional $1,000 of grant aid improves year-to- 
year persistence by 1.2 percentage points with smaller effects for degree completion. 

The rest of the paper is structured as follows. In the next section, we discuss our 
methodology, including the eligibility criteria, literature search, the coding of primary studies, 
and the analytic strategy. Then we present results from the main analysis, subgroup analyses, 
analysis of publication bias, and risk of bias assessment. We end with a discussion of our 
findings, their policy implications, the limitations of our study, and recommendations for future 
research. 

Method 

Our study is designed to examine the causal estimates of grant aid on postsecondary 
persistence and degree completion. To define the eligibility criteria, literature search, data 
analysis, and reporting conventions, we followed the Preferred Reporting Items for Systematic 
Reviews and Meta-Analysis standards as defined by Moher et al. (2009). 

Eligibility Criteria 

Primary studies eligible for inclusion in this meta-analysis need to meet the following 
criteria: (a) the sample is comprised of students in postsecondary education; (b) students are 
eligible for grant aid programs; (c) the study reports quantitative results of students’ 


postsecondary persistence or degree attainment; (d) the study provides plausibly causal estimates 
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of the effects of grant aid on persistence or degree attainment by employing experimental or 
quasi-experimental estimation strategies; and (e) the study provides linear probability estimates 
of the effects of grant aid. It should be noted that we retained studies that only provided odds 
ratios estimates of the causal effects of grant aid for the systematic review, but their point 
estimates are not comparable with linear probability estimates. Because they are focused on 
programs at the secondary school level, we also did not include studies that provide estimates of 
the effects of secondary school voucher programs or secondary school scholarships on 
postsecondary persistence and degree attainment (e.g., Chingos & Peterson, 2015). Lastly, we 
did not include studies on the effects of loans on persistence and attainment as loans are 
substantially different than grant aid because loans need to be repaid (e.g., Melguizo, Sanchez, & 
Valasco, 2016). 
Literature Search 

Given the topic of this review, we obtained primary studies from searching commonly 
used economic and general social science databases, including ERIC, WorldCat, ProQuest, 
JSTOR, NBER and EconLit (see Appendix Table 1 for a complete list of databases used). 
Through an iterative process of balancing an inclusive search string and a reasonable number of 
records that can be screened and analyzed thoroughly, we created the following search string: 
("need-based aid" OR "merit aid" OR "financial aid" OR grant OR scholarship OR "work- 
study") AND (persist* OR retention OR attrition OR graduat* OR dropout OR attain* OR 
degree), which returned over 21,000 studies. We also searched for “grey” literature using 
Dissertation and Thesis Repositories in WorldCat and ProQuest as well as a general Google 
search for evaluation reports of well-known financial aid programs such as the Florida Bright 


Futures, HOPE programs, or Pell Grant. In addition to searching databases, our literature search 
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also included an examination of reference lists uncovered through the process above and 
previous reviews of the financial aid literature (e.g. Angrist, Oreopoulos, & Williams, 2014; 
Deming & Dynarski, 2009). We also searched economic journals such as the Journal of Human 
Resources and policy analysis and evaluation journals such as the Journal of Policy Analysis and 
Management as they frequently publish articles evaluating the effects of financial aid on 
postsecondary outcomes. Our official search ended the first week of January 2018. We did not 
limit our search on publication date, location, or language. We focus our analyses on the effects 
of grant aid and postsecondary success in the United States instead of internationally as 
postsecondary grant aid functions differently in the United States than elsewhere. However, we 
also provide results for U.S. and international studies combined since the study of grant aid and 
postsecondary success is of scholarly and policy interest in the United States and abroad. 

Studies Meeting Eligibility Criteria. Starting with the results returned from our search 
of databases and previous reviews, we used a three-phase process to screen for primary studies 
that meet all eligibility criteria, as illustrated in Figure 1. First, two authors independently read 
the title, abstract, and introduction for 25 percent of the studies obtained in our original search 
using the search string. The two coders checked if there were studies that were retained by one 
coder but not the other and discussed reasoning for inclusion or exclusion. We split the 
remaining studies between those two authors for single-author review. We retained a study if the 
title, abstract or introduction mentioned that the study contained empirical results pertaining to 
causal estimates of grant aid and postsecondary persistence and attainment. Some examples of 
studies excluded in this phase included quantitative reports that did not provide causal estimates 
or studies that estimated the effects of grant aid on other postsecondary outcomes, such as 


enrollment or credits taken, that are distinct from persistence or degree completion (more details 
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about the measures of persistence and degree completion are provided below). In all, we 
screened over 21,000 studies. 

In phase two, we were left with 77 studies for full text reading and two coders 
independently assessed whether each study fit the eligibility criteria outlined above. The coders 
discussed any discrepancies and made exclusion decisions upon consensus. A third author 
resolved any disagreement. From these fully reviewed studies, we excluded studies that did not 
provide causal estimates of grant aid on postsecondary persistence or attainment. For multiple 
reports from the same study (e.g., a dissertation and corresponding journal article or reports from 
multiple years for the same evaluation), we kept only the most current publication. 

In phase three, we contacted authors to request information when eligible studies are 
missing key information. We sent e-mails to lead authors requesting information and re-sent 
these e-mails if we did not receive a response within four weeks. We excluded eligible studies if 
key information such as standard errors for effect estimates could neither be calculated nor 
obtained from the authors. If the standard error or the t statistic was not provided, but the 
significance level was indicated, we used a conservative estimate of the standard error by 
calculating the t statistics for the p value corresponding to reported significance levels. This is a 
conservative estimate of the standard error since it provides the largest standard error for a given 
significance level. Further details on how we calculated standard errors are included in the 
Analysis section below. We note that four studies that provided odds ratios estimates instead of 
linear probability estimates were excluded from the main quantitative analysis, but we discuss 
them in conjunction with the findings. At the end of phase three, we were left with a sample of 
47 primary studies, 42 primary U.S. studies and 5 international studies, which met all the 


eligibility criteria that provided the linear probability estimates of the effects of grant aid on 
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persistence and degree completion. This set of studies serves as our analytic sample for the meta- 
analysis. 
Coding Reports 

Two of the authors independently coded relevant information for each of the 47 eligible 
studies using a common coding schema (Appendix Table 2). A third author also coded a quarter 
of the studies using the same coding schema, and all three authors reviewed those studies to 
clarify any coding issues and resolve any discrepancies. We checked to see if all information was 
coded consistently and whether we selected the point estimates and standard errors from the 
same model if the primary authors provided multiple estimation models. Throughout the coding 
process, the two coders met monthly to compare codes for each study, updating the coding guide, 
and noting disagreements. Treating each cell of our coding matrix as an input, coder agreement 
occurred in 96% of the cells. Discrepancies among the remaining studies were resolved by 
consensus between the two coders, and the third author resolved any remaining disagreement not 
resolved between the first two coders. Next we describe relevant measures in greater detail. 

Dependent variable. Our main outcomes of interest are causal estimates of the effects of 
grant aid on postsecondary persistence and degree completion and the associated standard errors 
of those estimates (Lipsey & Wilson, 2001; for papers that employ similar methods, see 
Valentine, Konstantopoulos, and Goldrick-Rab (2017) and Holme, Richards, Jimerson, and 
Cohen (2010)). Persistence has two distinct categories: within-year persistence and year-to-year 
persistence. Within-year persistence is operationalized as students persisting from term to term in 
the same academic year. Year-to-year persistence is operationalized as students persisting from 
one academic year to a subsequent academic year, which includes first-to-second year 


persistence, second-to-third year persistence, third-to-fourth year persistence, and first-to-fourth 
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year persistence. If a study estimated multiple types of persistence, such as separate estimates for 
first-to-second year persistence and second-to-third year persistence, then we recorded each 
estimate separately (e.g., Clotfelter, Hemelt, & Ladd, 2016). 

Degree completion has two distinct categories: on-time attainment and delayed 
attainment. On-time attainment is considered to be degree completion within two years for two- 
year colleges and within four or five years for four-year colleges. Delayed attainment is three 
years or more for two-year colleges and six years or more for four-year colleges.’ Because our 
measures of both persistence and degree completion are binary, our primary studies contain a 
blend of linear probability models and odds ratios from logit models. The vast majority of studies 
employ linear probability models, which are interpreted as changes in the probability of 
persisting or degree attainment. These estimates are often discussed as percentage point changes 
in the probability of the outcome occurring, and we recorded each linear probability estimate as a 
percentage point change. 

Moderating variables. We coded a series of a priori moderators where we examined 
how the effects of grant aid varied by different financial aid program and study characteristics. 
These moderators were selected based on our reading of the literature and prior work we have 
conducted on financial aid. Specifically, we included the following variables as moderators: (a) 
the country in which the study took place; (b) the methods the study employs to obtain a causal 
estimate such as randomized control trial (RCT) or regression discontinuity design; (c) whether 
the study was peer reviewed; (d) eligibility type for the program (whether the grant aid program 


was need-based, merit-based, a combination of need and merit, or other); and (e) program 


'Tn the literature, on-time and delayed attainments are usually referred to as 100% and 150% time. For 
compatibility and ease of interpretation, we have also categorized 125% time as on-time attainment and greater than 
150% as delayed attainment. 
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characteristics such as additional treatments that might include additional academic or social 
supports. We expect postsecondary financial aid to function differently in the United States than 
other countries, particularly those in continental Europe where postsecondary tuition is either 
free or low cost. With regard to study quality, RCTs are the gold standard for establishing strong 
causal evidence of an intervention and regression discontinuity design has been shown to be a 
valid alternative to RCTs (Maas et al., 2017; Murnane & Willett, 2010; West et al., 2008). 
Furthermore, we recognize that there may be publication bias where only statistically significant 
results would be published, and the peer-review moderator allows us to explore the likelihood of 
such a bias in this context (Borenstein, Hedges, Higgins, & Rothstein, 2009; Rothstein, 2008). In 
terms of eligibility type, we expect the type of aid may have differential effects on persistence 
and degree completion since they have different requirements for award receipt, and they provide 
aid to different populations of students (Dynarski, 2004; Heller & Marin, 2002). Concerning 
program characteristics, we expect aid that has additional treatments, such as faculty advising or 
other academic supports, may help students acclimate to the college environment, thereby 
increasing persistence and degree attainment more than if the program offered grant aid alone 
(Angrist et al., 2014; Scrivener et al., 2015). 
Analytic Strategy 

Analysis of these data follow methods presented by Borenstein, Hedges, Higgins, & 
Rothstein (2009). Below, we describe analytical decisions in choosing between fixed-effect and 
random-effects models, selecting causal estimates, and assessing risk of bias from differences in 
study quality. 

One important choice for this meta-analysis was the decision between a fixed-effect 


versus a random-effects model. In the parlance of meta-analysis, a fixed effect meta-analysis 
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assumes all studies are estimating the same treatment effect whereas a random-effects model 
allows for differences in the treatment effect (Riley, Higgins, & Deeks, 2011). In other words, 
the fixed-effect model assumes a true effect size across all studies whereas a random-effects 
model allows the real treatment effect to vary across populations and programs (Borenstein, 
Hedges, Higgins, & Rothstein, 2009). Mechanically, the fixed-effect model assigns weights (W;) 


to each study (i) using the inverse of each within-study variance (V,, ): 


1 
Wi Fixed = a (1) 


In contrast, the random-effects model weights studies using both the within-study variance and 


the estimated between-study variance (T*): 


1 
2 
Vy, +T 


(2) 


Wi random = 


For this investigation, a random-effects model is most fitting because substantial 
variation exists across studies in terms of the requirements for aid eligibility, the amount of aid 
received, whether there were additional treatments, and whether students attended two-year or 
four-year institutions. Moreover, we do not expect the effects of grant aid to be homogenous 
across different populations and settings, particularly when the treatment dosage or the amount 
of aid provided varied from study to study. Additionally, we relied on heterogeneity statistics to 
inform our decision to use random-effects models. 

In terms of selecting the causal estimates, we primarily used the preferred estimates of 
the primary authors based on the authors’ explicit mention of their preferred estimates or their 
discussion of the estimates. If the primary authors did not have preferred estimates or if they 
preferred to provide all the estimates (along with their pros and cons) without emphasis, then we 


used our professional judgment and selected the most plausible causal estimates based on the 
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methods used and the extent to which they addressed internal validity issues of each estimation 
method. 

In terms of risk of bias or bias in the estimate of the treatment effect that can come from a 
number of sources, such as selection bias, attrition bias, or reporting bias, we chose to use an 
inclusive approach that will include all studies that satisfy our eligibility criteria. 
Understandably, this choice may introduce bias from poorly designed studies or studies of low 
quality. To address this concern, we took two separate approaches. In the first approach, we 
limited our analysis to study designs that provide strong causal evidence: randomized control 
trials and regression discontinuity designs (Maas et al., 2017; Murnane & Willett, 2010; West et 
al., 2008). In the second approach, we used the quality rating approach as suggested by Lipsey 
and Wilson (2001). In this approach, two coders independently rated each study holistically 
using our professional judgment and expertise in quantitative causal analysis of the quality of the 
study on a scale of 1 to 5 where 1 has high risk of bias and 5 has low risk of bias. For instance, to 
assess the internal validity of regression discontinuity studies we considered whether the 
researchers provided evidence of non-manipulation of the forcing variable and showed 
smoothness of the forcing variable around the threshold via the McCrary test, covariate balance 
checks on either side of the threshold, robustness of findings across various bandwidths, 
parametric and non-parametric specifications, and falsification tests. Appendix Table 3 contains 
the criteria we used to determine our rating. The two coders then discussed their rating and 
resolved any discrepancy by consensus. Any remaining disagreement was independently 
resolved by a third coder. We used these ratings as a robustness check to show that the findings 


from the main analyses are similar when restricted to only the high quality studies. 


* For more information on issues of causal inference for a variety of quasi-experimental designs, please refer to 
Shadish, Cook, and Campbell (2002) and Murnane and Willett (2010). 
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Results 

Table 1 presents descriptive information about the primary studies included in the meta- 
analysis separated into the full sample that includes international studies and U.S. only studies. 
Since postsecondary financial aid functions differently in the United States than other countries 
and the majority of the studies were from the United States, we focus our analysis and discussion 
on U.S. only studies although we do provide descriptive information in Table 1 and summary 
analysis for the full sample in Appendix Table 4, which includes U.S. and international studies. 

In terms of the U.S. only studies, all 42 studies are from 2004 to 2017 with the majority 
of the studies published in the last ten years. The majority of the studies, 83 percent, are 
considered to be high quality studies by our subjective rating. Sixty percent are RCTs or employ 
regression discontinuity. About half are published in peer-reviewed journals. The number of 
students in the studies ranges from a little over a hundred students to over 110,000 students, with 
an average of over 17,000 students. These studies investigated grant aid receipt that ranges from 
less than $300 to over $19,000 with an average of nearly $2,500. About a quarter of the studies 
have some form of additional treatment attached to the grant aid. About 21 percent of the studies 
are merit-only studies, 71 percent have some form of need-based component, and 8 percent have 
other eligibility requirements other than merit or need requirements such as veteran status, social 
security benefits, and having spent high school in a particular area (e.g. Barr, 2015; Bartik, 
Hershbein, & Lachowska, 2015; Ramsey, 2013;). Because some studies provide multiple 
outcome estimates, these 42 U.S. studies provide 9 within year persistence estimates, 25 year-to- 
year persistence estimates, 20 on-time degree completion, and 19 delayed completion effect 


estimates. These statistics are comparable to the full sample that includes international studies. 
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Table 2 presents the meta-analysis random-effects estimates of the causal effect of grant 
aid on postsecondary persistence and degree completion in the U.S. In terms of within year 
persistence, the summary estimate from nine studies indicates that grant aid increases within year 
persistence by 3.2 percentage points. Although the standard error is fairly sizable due to the 
limited number of studies providing within year estimates, this result is still statistically 
significant at the 5% level. Next we find grant aid increases year-to-year persistence by a modest 
1.8 percentage points with a fairly precise standard error of .005, which is highly statistically 
significant; the 95% confidence interval ranges from about one to three percentage points. In 
terms of degree completion, we find that grant aid increases on-time and delayed completion by 
2.4 percentage points and 2.5 percentage points respectively with reasonably precise standard 
errors. Overall, our results indicate that grant aid has a substantial impact on postsecondary 
persistence and degree completion in the U.S. These results comport with our expectations that 
aid can help students persist and succeed in postsecondary education. As noted previously, 
although we focus our discussion on U.S. studies, our U.S. and international studies results are 
very comparable to the U.S. only results discussed here (Appendix Table 4). 

As discussed previously, we rely on the random-effects meta-analysis model because we 
expect the effect of grant aid to vary by the amount of aid, the requirements, and the intended 
populations. However, we also present empirical evidence that random-effects models are more 
appropriate than fixed-effect models in Table 2. For each main outcome, we present a set of 
standard heterogeneity statistics of the study effects. For instance, in terms of on-time degree 
completion the percentage of observed variance across 20 studies that reflects true heterogeneity 
in effect sizes (1°) is 85.136, indicating that less than 15 percent of the total variation can be 


attributed to random error. The Cochrane’s Q statistic tests the null hypothesis of homogeneity 
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across studies, and with Pg at .002, we find strong evidence to reject the null hypothesis that the 
true dispersion of effect sizes is zero, providing further support that there is real heterogeneity of 
effects across studies. Together, these measures present empirical evidence there is heterogeneity 
in effect sizes, justifying the random-effects models, and furthermore, we find similar evidence 
of heterogeneity for all four outcomes. To explore these differences further, we examine the 
forest plots for all four outcomes, which provide further evidence that the effect estimates vary 
across different study and program characteristics. 

Figures 2 through 5 present forest plots for within year persistence, year-to-year 
persistence, on-time completion, and delayed completion respectively. For instance, Figure 3 
presents a forest plot of the overall random-effects model for year-to-year persistence. Each row 
represents an effect size from a primary study in our meta-analysis, plotted according to the size 
of the effect estimates. Furthermore, the effect sizes along with the 95% confidence intervals are 
provided for each study as well as the weight that each study contributes to the overall effect 
size. Studies that provide more precise estimates are given more weight. As such studies that 
provide imprecise estimates are given little weight in the overall estimate such as the Upton 
(2016) study that contributed only .01 percent to the overall estimate. The dotted vertical line 
intersecting the diamond at the bottom of the graph shows the average effect size across the year- 
to-year persistence studies. In other words, these forest plots provide detailed information about 
the effect estimate, the precision of the estimate, how much they contribute to the overall result, 
and how much the effects vary from study to study. In general, all four graphs indicate that there 
is substantial variation across studies in terms of the effects of aid on persistence and degree 
completion. To examine whether this heterogeneity can be explained by observable study 


characteristics, we turn to the moderator results. 
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Table 3 presents the results for within year and year-to-year persistence based on 
different study and program characteristics. We note that there is only limited evidence of 
moderator effects for within year persistence due to the small number of studies that analyze this 
outcome. As such, we focus our discussion on the moderator effects for the year-to-year 
persistence outcome, although most of the results also extend to within year persistence. For 
year-to-year persistence, we observe that the high quality studies and RCT/RD studies provide 
slightly higher point estimates than the overall effect estimate, although there is substantial 
overlap in the 95% confidence intervals. Estimates from peer-reviewed studies are also higher 
than the overall estimate, suggesting there may be publication bias, which we explore further 
below. In terms of additional treatment, we find that the effect size is nearly doubled suggesting 
that both aid and the other forms of support such as faculty advising, peer support, or academic 
support have substantial effects on postsecondary persistence. Lastly, we find that merit-only 
financial aid programs do not appear to affect persistence although the standard error is fairly 
sizable. This novel finding suggests that, although merit-aid programs may likely affect where a 
student chooses to go (Deming & Dynarski, 2009; Hossler et al., 2009), it does not appear to 
improve persistence, most likely because students are less likely to face financial obstacles to 
persistence. In contrast, grant aid programs that include a need-based component seem to have a 
substantial impact on persistence at 2.5 percentage points with a precise standard error of .005. In 
other words, our meta-analytical results indicate the 95% confidence interval of the effect of 
need-based grant aid program on year-to-year persistence ranges from 1.4 to 3.5 percentage 
points. Next we discuss the moderator results for degree completion. 

Similar to Table 3, Table 4 presents the moderator results for on-time and delayed degree 


completion. The estimates for the high quality and RCT/RD studies for both on-time and delayed 
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degree completion are comparable to the overall estimates. The estimate from peer-review 
studies is the same as the overall result for on-time completion and it is lower for the delayed 
completion. For delayed completion, this would indicate that the non-peer-reviewed studies’ 
estimates are higher than the peer-reviewed studies’ estimates, which we will discuss further in 
the next section. In terms of additional treatment, the estimates for both on-time and delayed 
completion for studies with additional treatment are more than 50 percent higher than the overall 
estimates, which reinforces our earlier results in Table 3 indicating that additional supports likely 
help students to persist and complete their degrees. In terms of merit- and need-based aid, for on- 
time completion we find that although their point estimates are the same and very comparable to 
the overall estimate, the merit only result is not statistically significant due to a large standard 
error. The need-based aid program has an estimated effect of raising on-time degree completion 
by 2.0 percentage points with a precise standard error of .005. For delayed completion, the point 
estimate for merit-only program is 1.9 percentage points but it is statistically insignificant. Need- 
based grant aid programs have an estimated effect of raising delayed degree completion by 2.4 
percentage points with a standard error of .008.° 

Our results thus far have shown that grant aid does indeed positively affect postsecondary 
persistence as well as degree completion. However, the above analysis ignores variation in aid 
amounts across programs and studies. Policymakers and enrollment managers are interested in 
understanding not only whether aid affects student outcomes but also how much aid is necessary 
to achieve a specific effect. For instance, previous literature has established that a $1,000 of aid 


approximately raises enrollment by three to four percentage points (Deming & Dynarski, 2009). 


3 We note only one study (Chen & Hossler, 2017) examined the effects of grant aid specifically for nontraditional 
students. Without the inclusion of this study, the estimated effect of raising delayed degree completion from 11 
studies is 3.1 percentage points with a standard error of .010. 
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Following this tradition, we also provide an estimation of the effect of $1,000 of aid on 
persistence and degree completion in Table 5 using OLS regression of each study’s treatment 
effect for the various outcomes on the average amount of aid provided in each study. We note 
our estimates are rough estimates due to the limited number of studies for each outcome. 
Furthermore, if the effect of aid is non-linear, our estimates will not be accurate.* Thus we 
provide the estimates with the caveat of the limitations of the number of primary studies that 
exist. 

Table 5 provides the effect of $1,000 of grant aid on within year and year-to-year 
persistence as well as on-time and delayed degree completion. Since there are only nine within- 
year studies, our OLS estimates are less precise due to the limited number of studies reporting 
this outcome. For the year-to-year persistence estimate, we find that $1,000 of aid increases year- 
to-year persistence by a statistically significant 1.2 percentage points. For degree completion, we 
find $1,000 of aid increases on-time and delayed degree completion by a statistically 
insignificant 0.8 percentage points. To illustrate these findings, Figure 6 shows the effect sizes 
from individual studies and the treatment effect per $1,000. 

Negative Effects and Gender Differences 

Even though the vast majority of the primary studies find positive or null effects of grant 
aid on persistence and degree attainment as illustrated in Figures 2 through 5, a few studies find 
there may have been detrimental effects of grant aid. We consider possible reasons why the 
findings from these particular studies are contrary to expectations and the other studies. For 
instance, Partridge (2013) found that the Bright Futures program reduced year-to-year 


persistence by about four percentage points. The author suggested that the negative impacts may 


4 We do not have sufficient degrees of freedom to explore non-linearities in these estimates via a polynomial 
regression function, but we believe this is an important avenue for future research. 
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be due to some students gaming the system such that academically weaker students, who should 
not have qualified to receive the merit-based grant, received the award anyway. These students 
were less likely to persist. The author did provide some evidence to suggest that there may have 
been some gaming of continuous variables such as high school GPA and SAT scores due to 
discontinuities around these variables at the cutoff points (Patridge, 2013). Using a regression 
discontinuity design, Cohodes and Goodman (2014) found that a Massachusetts merit aid 
decreased year-to-year persistence as well as on-time and delayed degree attainment. The authors 
argued and provided some evidence that the most plausible explanation for the negative effects 
of this aid program was that the aid induced students around the eligibility cutoff to attend 
cheaper and lower quality colleges with substantially lower graduation rates than they would 
have otherwise. The studies and the authors’ explanations of the negative impacts indicated that 
in some circumstances, there may be negative unintended consequences of aid that researchers 
and policymakers should consider, especially in regards to merit-based grants. 

In terms of heterogeneous treatment effects, several studies separated the results by 
gender to assess whether the effects are different for men and women, and several found that 
treatment effects varied by gender. For instance, Bartik, Hershbein, and Lachowska (2015), 
Evans and Nguyen (2017), and Zhang (2013) found the effects of aid may be more positive for 
women than men. For example, Evans and Nguyen (2017) found that an average increase of 
$1,100 in grant aid reduced weekly job hours by 1.5 to 2 hours for women and marginally 
increased their within-year persistence by 3.3 percentage points with null effects for men. On the 
other hand, Barr (2016) and Scott-Clayton (2011b) found the opposite. Barr (2016) found that 
men who received aid were 6.2 percentage points more likely to obtain their degree relative to 


men who did not receive the aid, compared to women who were 1.2 percentage points more 


THE EFFECTS OF GRANT AID 23 


likely to graduate. Some studies, however, found that the effects for men and women were not 
substantially different (Angrist, Lang, & Oreopoulos, 2006; Page, Castleman, & Sahadewo, 
2016; Scrivener et al., 2015). There are not enough such studies for us to estimate separate 
results by gender across the four outcomes in a formal meta-analysis. 
Publication Bias 

A common threat in meta-analyses is publication bias in which the literature included in 
the study study may be systematically unrepresentative of the true population of the completed 
studies. We consider this threat in the context of our analysis of the effects of grant aid on 
persistence and degree completion. To explore this threat, we include Figures 7 and 8 that show 
the contoured enhanced funnel plots for persistence and degree completion respectively. The 
contoured enhanced funnel plots are designed to help detect publication bias and other forms of 
bias by looking at the asymmetry of the plots. The contour shows if studies are missing in the 
areas of non-significance (the inner most funnel) or in the areas of higher statistical significance 
(the outer funnels). If studies are missing in the areas of non-significance, this suggests the 
possibility that the asymmetry is due to publication bias, meaning that non-significant results are 
not being published, or, alternatively, that there really is a statistically significant effect. Studies 
missing in the area of high significance suggest the cause of the asymmetry may be more likely 
due to other factors than publication bias such as study quality. Asymmetry to the left or right of 
the center of the funnel indicates that studies are systematically more likely to have found 
negative or positive results respectively. 

Figures 7 and 8 both provide no evidence that our analysis lacks studies with small effect 
estimates; we observe a substantial number of studies that have effect estimates that are non- 


significant. However, there are asymmetries in both figures due to a lack of negative effect 
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estimates because there are not many studies reporting negative results of the effects of aid on 
persistence and degree completion. While these plots suggest the possibility of bias in the non- 
publication of negative results, we argue that the extent of this bias is small. First, there is a 
concentration of studies around the zero effect estimates that are precisely measured. Second, our 
inclusive and exhaustive literature search process found non-journal publications such as 
working papers, research reports, and program evaluations, and the presence of these types of 
studies in our analytic sample alleviates the concern that insignificant or negative results are 
being systematically excluded. Third, we argue that large-scale studies are likely to be published 
regardless of whether they found negative or null effects given the substantial amount of money 
invested in these students and for the widespread interest among researchers and policy makers 
in higher education. For instance, we do find large RCTs and programs that found null or 
negative results (e.g., Anderson & Goldrick-Rab, 2016; Cohodes & Goodman, 2014; Scott- 
Clayton, 2011b). Fourth, we generally expect the effects of grant aid on postsecondary outcomes 
to be either null or positive, in accordance with the predictions of economic theory (Angrist et 
al., 2015). Consequently, we do not believe publication bias is a serious threat to our findings. 
Risk of bias 

We are also concerned with the possible risk of including studies with poor internal 
validity potentially biasing the results. We believe this risk is minimal because we find that the 
vast majority, 83 percent, of the studies of grant aid and postsecondary outcomes included in our 
analysis, are high quality studies or studies (Table 1). Tables 3 and 4 also show that the effect 
estimates when including only high quality studies or only RCT and RD studies are comparable 
to the overall estimates. In other words, our results do not change substantially when we drop the 


high risk of bias studies and retain only the most plausible causal estimates in our analysis. 
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Although we considered using a meta-regression approach, our sample size limitations 
across the four outcomes of interest preclude such an analysis. A sample size of at best 29 studies 
and at worst nine studies produces a limited opportunity to include study characteristics as 
independent variables in the analysis due to the small degrees of freedom. Given these concerns 
and limitations, we opted for subgroup analyses instead as suggested by Bartolucci and Hillegass 
(2010). 

Discussion 

The state of research in the field of higher education financial aid policy is strong. Unlike 
many areas of education policy, there is ample causal evidence of the efficacy of grant aid 
programs on a multitude of postsecondary outcomes. However, there has yet to be a 
comprehensive meta-analysis of these causal studies on persistence and degree completion 
outcomes, outcomes which are essential to the economic and civic success of our society. We 
provide the first such study by incorporating 73 effect estimates from 42 primary U.S. studies. 

A meta-analysis is particularly useful in this context because of the heterogeneity of grant 
aid program structures and student response. As noted, aid amounts of the studies included in our 
analysis range between less than $300 and more than $19,000. Additionally, a quarter of 
programs studied have supplementary, non-financial, treatments. For policymakers in a state or 
institution, it is difficult to base financial aid decisions on results from a single study conducted 
in another state or institution of a program with a different structure, aid amount, and additional 
academic supports. Our meta-analysis mitigates these issues by consolidating results across 
programs and reaching a conclusion about the effects of aid on persistence and degree 


completion more broadly. 
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Our results confirm that grant aid improves persistence and degree completion. Although 
the results are stronger for programs with additional, non-financial supports, our meta-analysis 
results provide clear evidence that providing grant aid improves within-year and year-to-year 
persistence as well as on-time and delayed degree completion. Averaging the effects over all of 
the studies provides point estimates of approximately two to three percentage point increases in 
the probability of persisting and completing a degree. These results are consistent over 
robustness checks that only include the highest quality studies and studies employing strong 
plausible causal estimates. Considering other features of heterogeneity across aid program, we 
also conclude that the effects are weaker for merit-based financial aid than for need-based 
financial aid. Assuming a linear relationship of aid amount and impact, we estimate an additional 
$1,000 of grant aid improves year-to-year persistence by 1.2 percentage points, with smaller 
effects for graduation outcomes. 

Limitations and Future Research 

In addition to the limitations of our analysis highlighted above, we could not include four 
studies we found met every inclusion criteria except for publishing results as linear probability 
models. These four studies only reported outcomes in terms of log-odds ratios from logit models, 
making direct comparisons with the bulk of extant studies challenging. Although we did not 
include those four studies in our quantitative meta-analysis results, these studies demonstrate 
similar results qualitatively. For example, Arendt (2016) found positive effects on year-to-year 
persistence and delayed degree completion; Davidson (2015) found positive effects on year-to- 
year persistence; Henry et al. (2004) found positive effects on on-time degree completion; and 
Zhang (2013) found positive results for on-time degree attainment. Hence, we do not believe the 


inability to fully include these studies in our meta-analysis estimates biases our conclusions. We 
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do suggest future research efforts in all areas of quantitative research examining binary outcomes 
endeavor to provide estimates from both linear probability models and logit models to ease 
subsequent meta-analytic comparisons. 

We have several suggestions for future research that would further advance the scholarly 
understanding and application of financial aid to improve postsecondary outcomes. First, we 
highly encourage more randomized control trials that have multiple treatment arms to explore the 
individual and joint effects of non-aid treatments such as faculty advising, peer support, and 
academic support and guidance along with the provision of grant aid. Individual studies and our 
meta-analytic results consistently show that these additional treatments are effective in helping 
students persist and graduate. Second, our analysis focused on the effects of grant aid as a 
necessity since very few studies consider the effects of other forms of financial aid, such as 
borrowing. We encourage additional work to uncover the effects of borrowing on postsecondary 
success. Since loans are not equivalent to grant aid in many ways (Boatman, Evans, & Soliz, 
2017), we would not expect that loans would have the same impact on student success. Third, 
there is preliminary and mixed evidence to suggesting that the effects of grant aid may not be 
uniform between men and women. In some instances, women responded more positively to men 
and vice versa. In other studies, the effects do not seem substantially different. As such, we 
suggest future research shows treatment results for men and women separately in subgroup 
analyses. Moreover, future research needs to examine the study and program characteristics, 
such as the size of the grant, the prestige of the grant, and how the grant is communicated to the 
recipients, all of which may influence these differential outcomes between men and women. 


Policy Implications 
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Our findings have several important implications for policymakers. Although the 
magnitude of the persistence effect is small relative to the effect sizes observed for the impact of 
grant aid on college enrollment, our findings do suggest that institutions, states, the federal 
government, and private scholarship funds will find returns on providing grant aid to students 
after they have initially enrolled. Specifically, institutions often consider front loading aid by 
providing a greater amount of grant aid in the first year of enrollment than in subsequent years. 
While this financial aid packaging policy may increase the probability of initial enrollment in the 
institution, it may come at the cost of persistence given that subsequent receipt of financial aid 
after the first year has a positive effect on year-to-year persistence. Our findings also suggest that 
supplementing grant aid with additional treatments, such as academic or social supports, will 
improve impacts. These types of interventions likely help students overcome financial, academic, 
or social obstacles to success, and our findings support continued investment in such programs. 
Finally, merit aid, specifically state and institutional programs (as the federal government 
focuses little on merit aid), seems to have lower impacts on persistence and degree completion, 
suggesting policymakers should consider shifting grant aid towards students based on financial 
need instead of subsidizing students who would likely succeed without additional support. 

In conclusion, the findings of this meta-analysis provide strong support for the continued 
extensive monetary investment in grant aid for postsecondary education. Students would 
experience worse outcomes without this investment, and expanding these financial supports 
would not only induce more students to attend postsecondary education but also increase their 


educational attainment. 
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Tables 


Table 1 
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Descriptive information on the primary studies by study and program characteristics 


Full sample 


United States 


Study characteristics 
Publication year 
High quality 
RCT/RD 
Peer review 
Average sample size 
Range of sample size 
Program characteristics 
Average aid receipt 
Range of aid receipt 
Added treatment 
Merit only 
Any need-based 
Other eligibility requirements 
Number of treatment effects 
Within year persistence 
Year-to-year persistence 
On-time completion 
Delayed completion 


Number of studies 


2004-2017 
81% 
62% 
55% 
17,551 students 
115-111,793 students 


$2,489 
$291-$19,354 
26% 
23% 
68% 
9% 


a 
Zo 
22 
20 
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2004-2017 
83% 
60% 
52% 
17,467 students 
115-111,793 students 


$2,530 
$291-$19,354 
26% 
21% 
71% 
8% 


9 
25 
20 
19 
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Note. All included studies use linear probability models. Some studies include multiple types of treatment effects. 
High quality studies include only studies with scores 3 or higher on our subjective ranking scale. RCT/RD includes 
only randomized control trials or regression discontinuity studies. Peer review includes only peer-reviewed studies. 
Additional treatment indicates the aid program includes other benefits to the students in addition to the grant aid. 
Merit only includes studies that are non-need-based financial aid programs. Any need-based includes all studies that 
have need-based components. Other eligibility requirements include veteran status, social security benefits, or 


having spent high school in a particular area. 
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Table 2 
Meta-analytic results of the effects of grant aid on postsecondary persistence and degree 
completion in the United States 


Outcomes Main effect estimates Heterogeneity of study effects 
N Effect Standard Lower Upper P Q Pg 
Estimate Error Bound Bound 
Persistence 
Within year 9 0.032 0.010 0.013 0.051 72.715 29.320 0.001 
Year-to-year 25 0.018 0.005 0.009 0.028 72.376 86.882 <.001 
Degree Completion 
On-time 20 0.024 0.008 0.008 0.039 85.136 127.828 0.002 
Delayed 19 0.025 0.007 0.012 0.038 81.216 95.827 <.001 


Note. Within year persistence includes studies that examine the term-to-term enrollment or within year effects of 
grant aid. Year-to-year includes studies that examine the persistence rate from one year to another, such as year one 
to year two, year two to year three, or year one to year four. On-time degree completion is the 100-125% degree 
completion, or four/five years and two years for four- and two-year institutions respectively. Delayed degree 
completion is the 150% or more degree completion, or six years or more for four-year institutions and three years or 
more for two-year institutions. Lower and upper bounds come from 95% confidence intervals. 
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Table 3 


Meta-analytic results of moderators of the effects of grant aid on postsecondary persistence in 


the United States 


Main effect estimates 


Model N_ Effect Standard Lower Upper 
Estimate Error Bound Bound 
Within year 

Overall 9 0.032 0.010 0.013 0.051 
High quality 8 0.037 0.011 0.015 0.059 
RCT/RD 7 ~~ 0.037 0.012 0.013 0.061 
Peer review 3 0.067 0.035 -0.001 0.136 
Added treatment 3 0.077 0.034 0.011 0.143 
Merit only 2 0.018 0.016 -0.013 0.049 
Any need-based 7 0.037 0.012 0.013 0.061 

Year-to-year 
Overall 25 ~=—0.018 0.005 0.009 0.028 
High quality 21 ~=—0.020 0.005 0.011 0.030 
RCT/RD 16 = 0.022 0.007. 0.008 0.036 
Peer review 15 -0.023 0.006 0.011 0.034 
Added treatment 6 0.032 0.014 0.004 0.060 
Merit only 6 0.000 0.013 -0.026 0.026 
Any need-based 18 0.025 0.005 0.014 0.035 


Note. High quality studies include only studies with scores 3 or higher on subjective ranking scale. RCT/RD 
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includes only randomized control trials or regression discontinuity studies. Peer review includes only peer-reviewed 
studies. Additional treatment indicates the aid program includes other benefits to the students in addition to the grant 
aid. Merit only includes studies that are non-need-based merit studies. Any need-based includes any study that has a 


need-based component. Lower and upper bounds come from 95% confidence intervals. 
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Table 4 
Meta-analytic results of moderators of the effects of grant aid on degree completion in the 
United States 
Main effect estimates 
Model N Effect Standard Lower Upper 
Estimate Error Bound Bound 
On-time completion 
Overall 20 ~=0.024 0.008 0.008 0.039 
High quality 18 = 0.023 0.008 0.007 0.038 
RCT/RD 130.025 0.010 0.006 0.044 
Peer review 10 =0.024 0.016 -0.006 0.055 
Added treatment 5 0.038 0.012 0.014 0.061 
Merit only 5 0.021 0.021 -0.020 0.062 
Any need-based 14 =—-0.020 0.005 0.010 0.030 
Delayed completion 
Overall 19 =0.025 0.007 0.012 0.038 
High quality 17. ~—-0.029 0.008 0.013 0.046 
RCT/RD 12. ~—-0.028 0.012 0.005 0.051 
Peer review 9 0.013 0.007. = -0.001 0.028 
Added treatment 6 0.048 0.017 0.015 0.082 
Merit only 5 0.019 0.018 -0.016 0.054 
Any need-based 12. (0.024 0.008 0.009 0.039 


Note. High quality studies include only studies with scores 3 or higher on subjective ranking scale. RCT/RD 
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includes only randomized control trials or regression discontinuity studies. Peer review includes only peer-reviewed 
studies. Additional treatment indicates the aid program includes other benefits to the students in addition to the grant 
aid. Merit only includes studies that are non-need-based merit studies. Any need-based includes any study that has a 


need-based component. Lower and upper bounds come from 95% confidence intervals. 
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Table 5 
The effect of $1,000 grant aid on various outcomes for U.S. studies 
Persistence Degree Completion 
(1) (3) (4) 
Within year Year-to-year On-time Delayed 
Grant aid per $1,000 0.013 0.012° 0.008 0.008 
(0.021) (0.005) (0.014) (0.005) 
_cons 0.017 0.004 0.025 0.027 
(0.037) (0.013) (0.041) (0.020) 
N 9 20 18 


Note. Not all studies provided a treatment contrast. Standard errors in parentheses 


*p <0.10, * p< 0.05, ™ p < 0.01 
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Appendix Tables 


Appendix Table 1 
Results by Database 


Database Results 
ProQuest (includes dissertation and theses) 7,034 
WorldCat (includes dissertation and theses) 4,074 
ERIC 3,104 
Taylor and Francis Online 2,060 
EconLit 1,391 
NBER 1,280 
JSTOR (abstract) 1,018 
Google scholar (abstract) 945 
Google scholar (title only) 252 
JSTOR 148 
Directory of Open Access Journal (DOAJ) 146 
Total 21,452 
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Appendix Table 2 
Coding Guide 
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Study and Program Characteristics 


Variable Description Level of measurement 
Id ID Number assigned to study Continuous 
Leadauth Name of lead author Nominal 
Title Title of paper Nominal 
Yearpub Year paper was published Continuous 
Pubtype Type of publication (academic journal, policy report, Nominal 
conference paper, etc.) 
RCT Randomized control trial indicator 0,1 
Rct_rd RCT or regression discontinuity indicator 0,1 
Peer review Is the study a peer-reviewed publication? 0,1 
Method Identification strategy (RD, DiD, IV, PSM, RCT) Nominal 
USA Is the study based in the U.S.? 0,1 
Otherctry Name of the country where the study was conducted if not Nominal 
the U.S. 
State Name of state (if U.S.=1) Nominal 
Outcome The dependent variable(s) of the study: within year Nominal 
persistence, first year persistence, year-to-year persistence, 
degree attainment (100%), degree attainment (150%) 
FA program Name of program/experiment Nominal 
Need_merit_other Type of aid: need-based, merit-based, need-based with Nominal 
merit component, other 
Add_treat Any additional treatment other than the aid amount Nominal 
Requirement Pre-test score equivalence between treatment and control Nominal 
Baseline_equivalence Baseline equivalence of control and treatment group 0,1 
indicator 
Sensi_robust_falsi Does the study provide sensitivity and robust estimates? Nominal 
Does it provide falsification test? 
Study_quality The author’s professional judgment of the study’s quality 1-5 
from 1-5 (1-poor, 3-average, 5-excellent) 
Summary_of_FA Qualitative note of the aid program Qualitative 
Misc_note Miscellaneous notes Qualitative 
Study Outcomes 
Variable Description Level of measurement 
Outcometype Within year persistence, year-to-year persistence, degree Nominal 


Inst_sector 
Gender 
Main_ana 
LOR/LPM 
Beta 


SE 

Tstat 

Samplesize 
Treatment_contrast 


Note 


attainment (on time, delayed) 

Institutional sector (2-year, 4-year, pooled) 

Gender indicator (male, female, pooled) 

The main estimate (not subgroup) indicator 

Logged odds ratios or linear probability model 
Regression coefficient of the causal estimate of aid on 
outcome 

The standard error of the beta 

T-statistics of the beta coefficient estimate 

Sample size of the estimate 

The dollar difference in aid receipt between treatment and 
control group 

Additional notes about the estimates or study 


Nominal 
Nominal 
0,1 
Nominal 
Continuous 


Continuous 
Continuous 
Continuous 
Continuous 


Qualitative 
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Appendix Table 3 

Quality Criteria for Assessing Risk of Bias 
Quality Rating Considerations 
Was the study a randomized control trial? 
Was implementation fidelity measured and adequately described, and what are the 
implications of implementation fidelity on outcomes? 
What are the relative strengths of the study design? 
Was the analytic approach adequately described, and what are the relative merits of the 
approach used? 
Was the comparison condition adequately described, and does the comparison group provide a 
reasonable counterfactual? 
Were threats to internal and external validity considered and addressed? 
Were findings robust to different analytical decisions and model specifications? 
Was baseline equivalence established between treatment and comparison groups? (This is 
unnecessary for some approaches such as the difference-in-difference design.) 
What sampling decisions were made by the authors and did the analytic sample present any 
concerns to internal or external validity? 

Note: Studies with a rating of three, four, or five out of five were considered low risk of bias. 
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Appendix Table 4 


Meta-analytic results of the effects of financial aid on postsecondary persistence and degree 
completion for the United States and international studies 


Outcomes Main effect estimates Heterogeneity of study effects 
N Effect Standard Lower Upper P Q Pg 
Estimate Error Bound — Bound 
Persistence 
Within year 9 0.032 0.010 0.013 0.051 72.715 29.320 0.001 
Year-to-year 29 0.019 0.004 0.010 0.027 69.202 90.916 <.001 
Degree Completion 
On-time 22 0.025 0.007 0.012 0.038 84.170 132.663 <.001 
Delayed 20 0.026 0.006 0.014 0.039 80.744 98.672 <.001 


Note. Within year persistence includes studies that examine the term-to-term enrollment or within year effects of 
financial aid. Year-to-year includes studies that examine the persistence rate from one year to another, such as year 
one to year two, year two to year three, or year one to year four. On-time degree completion is the 100-125% degree 
completion, or four/five years and two years for four- and two-year institutions respectively. Delayed degree 


completion is the 150% or more degree completion, or six years or more for four-year institutions and three years or 
more for two-year institutions. 
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Figures 


Records identified through Additional records identified 
database searching through other sources 
(n = 21,452) (n= 18) 


Identification 


Screening 


Records screened Records excluded 
(n= 21,470) (n= 21,393) 


Full-text articles Full-text articles excluded, 
assessed for eligibility with reasons 
(n= 77) (n=8, outcomes not 
persistence or completion) 
(n=7, not causal estimate) 


Eligibility 


(n=6, miscellaneous reasons) 
(n=5, early draft of final 
studies already included) 
(n=4, odds ratios estimates, 
not linear probability model; 
retained for qualitative 
synthesis) 


Studies included in 
quantitative synthesis 
(meta-analysis) 

(n = 47) 


Figure 1. Flow diagram depicting the literature screening process resulting in the final sample of 
primary studies included in the quantitative analysis. Adapted from Moher et al. (2009). 
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Forest Plot: Grant Aid and Within Year Persistence 
Random Effects Model 


Study % 

ID ES (95% Cl) Weight 
Park & Scott-Clayton (2017) -0.02 (-0.06, 0.02) 10.09 
Partridge (2013) 0.01 (-0.01, 0.03) 14.46 
Evans & Nguyen (2017) 0.02 (-0.01, 0.05) 12.38 
Goldrick-Rab et al. (2016) 0.02 (0.01, 0.04) 16.06 
Richburg-Hayes et al. (2015) 0.04 (0.02, 0.06) 14.32 
Castleman (2014) 0.04 (-0.00, 0.08) 9.35 
Castleman & Long (2016) 0.04 (-0.00, 0.09) 8.51 
Patel & Valenzuela (2013) 0.05 (-0.00, 0.10) 7.81 
Barrow et al. (2014) _———— 0.15 (0.09, 0.21) 7.03 
Overall (l-squared = 72.7%, p = 0.000) 0.03 (0.01, 0.05) 100.00 
NOTE: Weights are from random effects analysis 


T T T T 
“4 “2 0 2 4 


Weights are from random effects analysis. ES is the effect size. Cl is the confidence interval. 


Figure 2. Forest plot for overall effect estimates of grant aid on within year persistence from 
primary U.S. studies. 
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Forest Plot: Grant Aid and Year-to-year Persistence 
Random Effects Model 


% 
ES (95% Cl) Weight 


Partridge (2013) -0.04 (-0.06, -0.02) 6.00 
Cohodes & Goodman (2014) -0.02 (-0.04, -0.00) 6.33 
Denning, Marx, & Turner (2017) 0.00 (-0.02, 0.02) 6.33 
Park & Scott-Clayton (2017) 0.00 (-0.05, 0.05) 2.49 
Clotfelter, Hemelt, & Ladd (2016) 0.00 (-0.04, 0.05) 3.16 
Alon (2011) 0.00 (0.00, 0.01) 7.93 
Anderson & Goldrick-Rab (2016) 0.01 (-0.05, 0.06) 2.36 
Mayer, Patel, & Gutierrez (2016) 0.01 (-0.03, 0.04) 4.21 
Marx & Turner (2017) 0.01 (-0.02, 0.04) 4.39 
Ramsey (2013) 0.01 (-0.06, 0.08) 1.67 
Bettinger (2015) 0.02 (0.01, 0.04) 6.90 
Doyle, Lee, & Nguyen (2017) 0.02 (-0.03, 0.07) 2.77 
Richburg-Hayes et al. (2015) 0.02 (-0.00, 0.05) 5.15 
Goldrick-Rab et al. (2016) 0.02 (-0.00, 0.05) 4.75 
Castleman & Long (2016) 0.03 (-0.02, 0.07) 2.93 
Carruthers & Ozek (2016) 0.03 (0.00, 0.06) 4.88 
Castleman (2014) 0.04 (-0.01, 0.08) 3.13 
Bettinger (2004 0.04 (0.02, 0.05) 6.33 
Chen & Hossler (2017) 0.04 (0.02, 0.07) 5.22 
Schudde & Scott-Clayton (2016) 0.05 (0.02, 0.07) 5.15 
Angrist, Autor, Hudson, & Pallais (2014) 0.05 (0.01, 0.08) 4.14 
Page et al. (2017) 0.06 (-0.05,0.17) 0.68 
Villarreal (2017: 0.07 (0.01, 0.13) 1.78 
Barrow et al. (2014) 0.12 (0.04, 0.20) 1.26 
Upton (2016) 0.19 (-0.66, 1.05) 0.01 
Overall (I-squared = 72.4%, p = 0.000) 0.02 (0.01, 0.03) 100.00 


NOTE: Weights are from random effects analysis 
-.4 “2 0 2 4 


Weights are from random effects analysis. ES is the effect size. Cl is the confidence interval. 


Figure 3. Forest plot for overall effect estimates of grant aid on year-to-year persistence from 
primary U.S. studies. 


THE EFFECTS OF GRANT AID o1 


Forest Plot: Grant Aid and On-time Completion 
Random Effects Model 


% 
ES (95% Cl) Weight 


Scott-Clayton (b) (2011) -0.14 (-0.36, 0.08) 
Cohodes & Goodman (2014) -0.02 (-0.04, 0.00) 
Castleman & Long (2016) 0.00 (-0.02, 0.03) 
Carruthers & Ozek (2016) 0.01 (-0.03, 0.04) 
Liu (2017) 0.01 (-0.00, 0.01) 
Park & Scott-Clayton (2017) 0.01 (-0.03, 0.05) 
Castleman (2014) 0.01 (-0.02, 0.05) 
Bartik, Hershbein, & Lachowska (2015) 0.02 (-0.04, 0.07) 
Denning (2016) 0.02 (0.01, 0.03) 

Villarreal (2017) 0.02 (-0.02, 0.06) 
Denning, Marx, & Turner (2017) 0.03 (0.01, 0.05) 


Castleman, Long, & Mabel (2017) 0.03 (-0.00, 0.06) 


Mayer, Patel, & Gutierrez (2016) 0.03 (0.00, 0.07) 

Gershenfeld (2016) 0.05 (-0.02, 0.11) 
Goldrick-Rab et al. (2016) 0.05 (0.00, 0.09) 

Clotfelter, Hemelt, & Ladd (2016) 0.06 (-0.03, 0.14) 

Scrivener et al. (2015) 0.06 (0.02, 0.10) 
Scott-Clayton (a) (2011) 0.07 (0.06, 0.08) 

Page et al. (2017) 0.08 (-0.05, 0.21) 1.17 
Toutkoushian et al. (2013) 0.50 (-0.01, 1.00) 0.09 
Overall (I-squared = 85.1%, p = 0.000) 0.02 (0.01, 0.04) 100.00 


NOTE: Weights are from random effects analysis 


“4 “2 0 2 4 


Weights are from random effects analysis. ES is the effect size. Cl is the confidence interval. 


Figure 4. Forest plot for overall effect estimates of grant aid on on-time completion from primary 
USS. studies. 
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Forest Plot: Grant Aid and Delayed Completion 
Random Effects Model 


ES (95% Cl) 


Anderson & Goldrick-Rab (2016) -0.04 (-0.10, 0.01) 
Cohodes & Goodman (2014) -0.03 (-0.04, -0.01) 
Schudde & Scott-Clayton (2016) -0.01 (-0.02, -0.00) 
Chen & Hossler (2017) 0.01 (0.00, 0.02) 
Welch (2014) 0.01 (-0.04, 0.07) 
Andrews, Imberman & Lovenheim (2016) 0.02 (0.01, 0.02) 
Allin (2015) 0.03 (0.00, 0.05) 
Clotfelter, Hemelt, & Ladd (2016) 0.03 (-0.05, 0.11) 
Castleman (2014) 0.04 (-0.01, 0.08) 
Mayer, Patel, & Gutierrez (2016) 0.04 (0.00, 0.07) 
Pantal (2006) 0.04 (-0.00, 0.09) 
Castleman & Long (2016) 0.05 (0.01, 0.09) 
Denning, Marx, & Turner (2017) 0.05 (0.01, 0.08) 
Barr (2015) 0.05 (0.02, 0.09) 


Villarreal (2017) 0.06 (-0.02, 0.14) 


Bettinger et al. (2016) 0.09 (0.04, 0.14) 


Bartik, Hershbein, & Lachowska (2015) 0.10 (0.01, 0.19) 
Scrivener et al. (2015) 0.18 (0.07, 0.29) 
Page et al. (2017) 0.19 (-0.02, 0.40) 0.36 


Overall (I-squared = 81.2%, p = 0.000) 0.03 (0.01, 0.04) 100.00 


NOTE: Weights are from random effects analysis 
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Weights are from random effects analysis. ES is the effect size. Cl is the confidence interval. 


Figure 5. Forest plot for overall effect estimates of grant aid on delayed completion from primary 
studies. 
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Figure 6: Grant aid amount plotted against effect size, percentage point increase per $1,000 of 
grant aid for two outcomes 
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Figure 7. Contoured enhanced funnel plots of grant aid and persistence from primary U.S. 


studies. 
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Figure 8. Contoured enhanced funnel plots of grant aid and degree completion from primary U.S. 
studies. 


